Skip to content

[WIP] Make ImageFiltering.imfilter pipeline differentiable#21

Open
SomTambe wants to merge 4 commits intomainfrom
som/kernels
Open

[WIP] Make ImageFiltering.imfilter pipeline differentiable#21
SomTambe wants to merge 4 commits intomainfrom
som/kernels

Conversation

@SomTambe
Copy link
Member

@SomTambe SomTambe commented Aug 25, 2021 •
edited
Loading

This PR aims to make the ImageFiltering.imfilter function differentiable.

  • Add adjoints
  • Get the gradient of the kernel
  • Put tests

The current issue I am facing causes the gradient of the kernel to be nothing. There seems to be a place where I have incorrectly put a NoTangent(), or a @nograd. Infact upon some further tracing I found out it does go inside the rule of __imfilter_inbounds!, but never calls the pullback inside it.

julia> img = testimage("cameraman");

julia> gradf(x, k, args...) = Zygote.gradient((a, b)->imfilter(a, b, args...) |> sum |> (z->z.val), x, k)
gradf (generic function with 1 method)

julia> gradf(img, t*t') 
(Tangent{Gray{Float64}, NamedTuple{(:val,), Tuple{Float64}}}[Tangent{Gray{Float64}}(val = 1.0,) Tangent{Gray{Float64}}(val = 1.0,) … Tangent{Gray{Float64}}(val = 1.0,) Tangent{Gray{Float64}}(val = 1.0,); Tangent{Gray{Float64}}(val = 1.0,) Tangent{Gray{Float64}}(val = 1.0,) … Tangent{Gray{Float64}}(val = 1.0,) Tangent{Gray{Float64}}(val = 1.0,); … ; Tangent{Gray{Float64}}(val = 1.0,) Tangent{Gray{Float64}}(val = 1.0,) … Tangent{Gray{Float64}}(val = 1.0,) Tangent{Gray{Float64}}(val = 1.0,); Tangent{Gray{Float64}}(val = 1.0,) Tangent{Gray{Float64}}(val = 1.0,) … Tangent{Gray{Float64}}(val = 1.0,) Tangent{Gray{Float64}}(val = 1.0,)], nothing)

The img gradient is incorrect too, and I shall get it rectified when I write tests and figure out the kernel thing.

@codecov-commenter
Copy link

codecov-commenter commented Aug 25, 2021

Codecov Report

Merging #21 (cf6fbf8) into main (890e927) will decrease coverage by 32.67%.
The diff coverage is 0.00%.

Impacted file tree graph

@@             Coverage Diff             @@
##             main      #21       +/-   ##
===========================================
- Coverage   88.23%   55.55%   -32.68%     
===========================================
  Files           3        4        +1     
  Lines         102      162       +60     
===========================================
  Hits           90       90               
- Misses         12       72       +60
Impacted Files Coverage Δ
src/filters/adjoints.jl 0.00% <0.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 890e927...cf6fbf8. Read the comment docs.

@DhairyaLGandhi
Copy link
Member

DhairyaLGandhi commented Aug 26, 2021

Yeah you definitely want the Tangent to wrap an array and not elements I would imagine.

DhairyaLGandhi
DhairyaLGandhi reviewed Aug 26, 2021
View reviewed changes
Copy link
Member

@DhairyaLGandhi DhairyaLGandhi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for taking this up! Looking forward to the tests. I'll also take a look through the adjoints to see if they're correct

src/filters/adjoints.jl
z)
y = __imfilter_inbounds!(r, out, A, kern, border, R, z)
function ∇__imfilter_inbounds!(Δy)
# ∇out should not have any gradients
Copy link
Member

@DhairyaLGandhi DhairyaLGandhi Aug 26, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since it will be used further down in the computation pipeline, i think you still need to provide it the gradient so it can be accumulated properly. Else the answer would be incorrect

src/filters/adjoints.jl
y = imfilter!(out, img, kernel, border, alg)
function ∇imfilter!_try(Δy)
k = default_resource(alg_defaults(alg, out, kernel))
ret = imfilter!(k, out, img, kernel, border)
Copy link
Member

@DhairyaLGandhi DhairyaLGandhi Aug 26, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this true generically?

@johnnychen94 johnnychen94 mentioned this pull request Nov 1, 2021
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@DhairyaLGandhi DhairyaLGandhi DhairyaLGandhi left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@SomTambe @codecov-commenter @DhairyaLGandhi